仅使用前视投影 得到的效果更好?可能是网络的规模不够大

尝试仅使用前视投影+sn


删除前视投影 仅保留pc分支:

loss非常不稳定 生成图像效果差


将pc分支的浅层特征经过down和res分支

另外将pc特征加入到discriminator中 或者从discriminator中取消投影输入


SN: spectral norm


在discriminator中删除conditional输入之后, feature matching loss和vgg loss的初始值变得非常高,这可能是因为没有conditional输入之后,discriminator在开始接受到的输入就完全是混乱的初始生成图像 没有与真实图像的feature比较类似的条件输入;但是loss下降的趋势还比较稳定(即没有造成训练过程的不稳定),后期也下降到和有conditional输入的discriminator一样的情况了


最终的测试结果没有加入conditional输入好


目前来看Generator输入的上采样图像的作用比较大


静止场景:

0016 0017 0019(步行街场景 行人多 光照复杂)


尝试noVGG 效果很差


尝试pixel shuffle 解决棋盘格效应


object sensitive loss


尝试对gamma进行随机增强,生成的图像亮度更高,在50轮的时候视觉效果更好一些,但是没有最开始的点云分支好


尝试加入pointnet2的head, 收敛速度变慢, 训练集有一些图像效果还可以,对训练集拟合效果不错


激光雷达+图像 去阴影


和相机重建出来的对比


将测试集换为tracking与object的差集 效果不错


尝试在resblock前加入pc特征 效果比resblock前加入的细节和边缘要好


对点云的intensity进行归一化


在Discriminator中取消SN后面的BN之后 Gan的Feature matching loss的量级下降了很大 从[7,12]下降到了[1, 2], 这可能是因为SN实现了Lipschitz连续条件 但是只使用SN时拟合效果非常差


无法复现之前pix2pix的效果了 目前的改动:Gamma:无效 flip:无效 batchsize 有一些效果,现在感觉可能是数据集的问题


尝试直接用前视投影的点云提特征?


加入膨胀卷积之后效果很好


远处的弱化 从loss上考虑



LiCAM segmentation 实验

Accumulating evaluation results...                                     │tmpfs      38G 3.8G  34G 10% /run

DONE (t=0.06s).                                              │/dev/sda2    439G 393G  24G 95% /

 Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.590              │tmpfs      189G 736M 188G  1% /dev/shm

 Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.882              │tmpfs      5.0M   0 5.0M  0% /run/lock

 Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.646              │tmpfs      189G   0 189G  0% /sys/fs/cgroup

 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.402              │/dev/sda1    511M 3.7M 508M  1% /boot/efi

 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.689              │tmpfs      38G  28K  38G  1% /run/user/108

 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.866              │tmpfs      38G  22G  16G 58% /run/user/1000

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.180              │tmpfs      38G   0  38G  0% /run/user/1001

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.613              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.625              │[sudo] password for bdbc201: 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.453              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.715              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907    



LiCAM 测真实图像

DONE (t=0.20s).                                              │/dev/sda2    439G 393G  24G 95% /

 Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.613              │tmpfs      189G 736M 188G  1% /dev/shm

 Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.885              │tmpfs      5.0M   0 5.0M  0% /run/lock

 Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.689              │tmpfs      189G   0 189G  0% /sys/fs/cgroup

 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.397              │/dev/sda1    511M 3.7M 508M  1% /boot/efi

 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.737              │tmpfs      38G  28K  38G  1% /run/user/108

 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.886              │tmpfs      38G  22G  16G 58% /run/user/1000

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.195              │tmpfs      38G   0  38G  0% /run/user/1001

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.646              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.662              │[sudo] password for bdbc201: 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.458              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.778              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.932



真实图像测SPADE

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.449

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.678

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.485

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.783

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.161

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.481

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.481

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.254

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.610

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.816


真实图像测皮鞋pix2pixHD

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.146

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.301

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.130

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.179

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.407

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.092

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.168

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.168

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.043

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.209

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.442


真实图像测LiCAM

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.388 

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.636 

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.407 

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.151 

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.511 

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.809 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.152 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.429 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.431 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.553 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.847 



真实图像测SPADE-LICAM-Intensity

直接High res

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.371

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.586

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.410

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.111

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.502

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.813

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.141

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.402

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.402

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.137

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.858


Low res 2 High res

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.365

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.583

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.399

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.112

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.791

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.142

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.398

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.398

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.138

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.530

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.829


目前看应该是需要平滑的conditional input(这个地方可以abalation study)


真实图像测SAPDE-LICAM

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.337

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.550

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.362

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.102

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.453

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.767

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.135

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.376

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.376

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.814


真实图像测pix2pix

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.007 

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.010 

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.010 

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004 

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.015 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.014 


真实图像测cycle

None 没有结果



改用OMP多核实现之后 生成1M个点可以从90s提高到10s


使用光栅化过程的zbuffer解决了深度遮挡的问题 可以考虑不用光线追踪 但是仍然是串行实现的


ray tracng 之后的结果会在属于同一个类别的大片联通区域产生孔洞,孔洞中的数值是与该区域的类别非常相似的数值。这在图像中无所谓,


闭运算前


闭运算后